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PREFACE 


There  is  a  growing  literature  and  interest  in  methods  for 
quantifying  subjective  judgments.  Several  ongoing  Air  Force  efforts 
utilizing  subjective  judgment  have  come  to  the  authors'  attention. 
Mission  Area  Analysis  requires  subjective  estimates  of  a  large  number 
of  parameters.  Long-range  planning  repeatedly  draws  on  judgments 
about  the  future  importance  and  worth  of  plans  and  geographical 
areas.  The  Constant  Quest  project,  directed  by  the  Readiness/NATO 
Coordination  Board,  highlighted  the  importance  of  subjective  judgments 
in  evaluating  command  and  control  systems. 

Thomas  Saaty  of  the  University  of  Pennsylvania  has  advanced  a 
popular  tool  for  quantifying  and  scaling  the  worth  of  a  set  of  objects 
or  entities.  For  problems  that  fit  the  Saaty  framework,  this  report 
details  an  improvement,  on  oaaly's  "eigeuvecLoi"  Leiuuique  (Refs.  [19] 
to  [30])  that  is  easier  to  use  and  more  amenable  to  statistical 
inferences. 

This  report  was  prepared  under  the  Project  AIR  FORCE  research 
study  effort,  "Evolving  Concepts  for  Long-Range  Planning." 
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SUMMARY 


The  recurring  need  to  utilize  subjective  information  and 
judgments  in  quantitative  analyses  has  been  widely  recognized  and  has 
been  treated  in  a  number  of  ways.  This  report  considers  the  problem 
of  using  subjective  paired  comparisons  to  estimate  the  relative  worth 
of  each  member  of  a  collection  of  objects  or  alternatives. 


Let  {Ej ,E„ , . . . ,Efl}  be  a  collection  of  objects  or  entities  that 


are  in  some  sense  comparable.  The  E^  may  be  alternative  plans  to 


achieve  some  goal,  alternative  objects  that  have  some  comparable 
utility,  or  generally  a  collection  of  entities  that  have  varying 
degrees  of  some  common  value.  A  vector  (u^ ,u„ , . . . ,u  )  is  called 


a  ratio  scale  for  the  collection  if  for  each  i  and  j,  u./u.  is  the 

i  j 


ratio  of  the  value  of  E^  to  the  value  of  E ^ .  For  example,  u^/ u ,. 


is  4  if  E^  has  four  times  the  value  of  . 


In  this  case  E,.  has 


1/4  the  value  of  E^  and  uj/u2  W- 

An  important  application  of  ratio  scales  is  in  the  study  of 
hierarchies.  A  hierarchy  is  a.  collection  of  objects  organized  into 
levels.  Suppose  that  for  each  level  there  is  a  ratio  scale  for  the 
objects  at  that  level  relative  to  any  object  at  the  next  level  up. 

The  ratio  scales  for  various  levels  can  be  combined  multiplicatively 
to  give  a  view  of  the  entire  hierarchy.  Because  hierarchies  are  used 
to  model  complex  systems  in  many  important  military  and  industrial 
applications,  the  estimation  of  ratio  scales  deserves  considerable 
attention. 

Suppose  that  a  ratio  scale  (u^  ,1^ , .  ■  •  ,un)  for  objects  E^E^, 


exists  but  is  not  known.  Let  a.  .,  i,j  =  l,2,...,n  be  subjective 

1  *  J 


estimates  of  u^/uj  made  by  a  judge.  In  particular,  a^  ^  =  1  for 


each  i,  and  a.  ,  =  1/a.  ..  The  matrix  A  =  [a.  . ]  of  subjective 
J.i  i.J  i,J  J 

pairwise  comparisons  is  called  a  judgment  matrix. 


If  the  judge  is  perfectly  consistent  in  making  estimates,  then 
the  matrix  A  will  satisfy  the  consistency  criterion 


p*» 


J 


vi 

ai  j*j  k  *  *i  k  for  (ac^  A*j»k  • 

If  this  condition  is  net,  then  any  coluan  of  the  matrix  A  gives  a 
rstio  scale  for  However,  judgments  are  frequently 

inconsistent,  and  many  judgment  matrices  do  not  satisfy  the  con¬ 
sistency  criterion.  A  mathematical  procedure  is  required  for  esti¬ 
mating  an  underlying  ratio  scale  based  on  an  inconsistent  judgment 
matrix  A. 

A  procedure  proposed  by  Thomas  Saaty  has  been  used  in  a  variety 
of  applications  calling  for  the  estimation  of  ratio  scales  from 
judgment  matrices.  Saaty  has  shown  that  corresponding  to  any  judgment 
matrix  is  s  positive  eigenvalue  exceeding  all  the  other  eigenvalues  in 
absolute  value,  and  the  normalized  eigenvector  corresponding  to  this 
maximal  eigenvalue  has  strictly  positive  components .  Because  the 
normalized  eigenvector  corresponding  to  the  unique  nonzero  eigen¬ 
value  of  a  consistent  judgment  matrix  does  give  a  ratio  scale  for 
the  matrix,  and  because  the  components  of  the  eigenvector  depend 
continuously  on  the  matrix  entries,  Saaty  uses  the  normalized  eigen¬ 
vector  corresponding  to  the  maximal  eigenvalue  as  an  estimate  for 
the  ratio  scale  underlying  any  judgment  matrix. 

The  normalized  eigenvector  corresponding  to  the  maximal  eigen¬ 
value  of  a  judgment  matrix  is  not  the  only  vector  that  gives  the 
underlying  ratio  scale  when  the  matrix  is  consistent.  The  underlying 
ratio  scales  in  the  consistent  case  are  also  given  by  any  column  of 
the  matrix,  by  the  vector  of  row  sums,  and  by  the  vector  of  row 
geometric  means  (nth  root  of  the  product  of  row  elements).  In  fact, 
if  the  judgment  matrix  is  consistent,  all  these  vectors  are  scalar 
multiples  of  the  normalised  eigenvector.  Moreover,  each  of  these 
vectors  depends  continuously  on  the  matrix  entries.  Thus,  the  argu¬ 
ment  in  favor  of  the  eigenvector  as  a  ratio  scale  estimator  holds  for 
several  other  vectors  as  well. 


11m  pnbla  of  estimating  ratio  scales  from  Judgment  matrices 
can  be  viewed  in  a  statistical  framework  as  follows.  Let 

.  taa)  be  a  ratio  scale  for  a  collection  |EB}» 

and  let  A  ■  (a^  j|  be  a  judgment  matrix  derived  from  pairwise 
comparisons  of  the  K^.  The  elements  a^ j  can  be  thongbt  of  as 
arising  from  ratios  of  the  scale  elements  bp  multiplicative 
perturbations—  i.e. , 

\.i  *  <VjK.j  •  » 

where  the  e^  .  sen  positive  random  variables.  Taking  logarithms 


ta  *i.j  •  ♦  «i.j  •  *•» 

\ 

share  bj  •  lm  n^  end  ijj  ■  In  Assuming  that  the  d^j  are 

independent  random  variables  with  aero  means  and  equal  variances, 
the  bast  linear  unbiased  estimator  of  tbs  b^  are  the  least  squares 
estimators^  given  bp 


Under  the  additional  assumption  that  the  d.  .  are  normally  distrib- 

/\  1  >  J 

uted,  the  u.  are  maximum  likelihood  estimators  of  the  u.. 

As  an  estimator  of  ratio  scales,  the  geometric  mean  vector  is 
preferable  to  the  dominant  eigenvector  in  several  respects.  First, 
the  geometric  mean  vector  is  easy  to  calculate,  even  by  hand;  cal¬ 
culation  of  the  eigenvector  usually  requires  an  iterative  procedure 
that  would  be  difficult  to  carry  out  by  hand  and  can  be  time-consuming 
even  on  a  computer.  Also,  because  the  geometric  mean  vector  arises 
from  a  well-known  statistical  model,  it  lends  itself  to  confidence 
interval  estimation  and  tests  of  hypotheses. 

Finally,  empirical  evidence  based  on  Monte  Carlo  experiments 
indicates  that  the  geometric  mean  vector  is  statistically  better  than 
the  dominant  eigenvector  under  two  quite  different  assumptions  on  the 
distribution  of  the  d.  ..  In  both  cases  the  relative  efficiency  of 
the  two  procedures  is  nearly  one  when  the  variances  of  the  d^  ^  are 
small,  and  the  geometric  mean  becomes  preferable  as  the  matrices 
become  increasingly  inconsistent. 
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I.  INTRODUCTION 


Over  the  past  three  decades,  military  and  industrial  researchers 
have  directed  considerable  effort  to  the  quantitative  analysis  of  sub¬ 
jective  data.  Analytic  tools  building  upon  the  subjective  judgments 
of  experts  have  been  used  in  such  diverse  fields  as  energy  policy 
analysis,  marketing  research,  economic  forecasting,  and  military  plan¬ 
ning.  Problems  amenable  to  the  analysis  of  subjective  information 
abound,  and  numerous  methods  have  been  proposed  for  acquiring  and 
treating  judgmental  data. 

One  question  that  arises  in  the  treatment  of  subjective  data  is 
how  to  construct  a  scale  of  relative  merit  for  a  collection  of  objects 
or  activities  based  upon  subjective  comparisons  of  each  pair  in  the 
collection.  For  example,  consider  a  collection  of  three  objects, 
labeled  A,  B,  and  C.  Suppose  that  an  expert  believes  A  has  twice  the 
merit  of  B,  B  has  three  times  the  merit  of  C,  and  A  has  six  times  the 
merit  of  C.  It  is  natural  to  construct  a  scale  of  relative  merit  for 
A,  B,  and  C  as  (1,  1/2,  1/6).  However,  suppose  that  the  expert  says  A 
has  twice  the  merit  of  B  and  B  has  three  times  the  merit  of  C,  but  A 
has  only  four  times  the  merit  of  C.  In  this  case,  it  is  not  so  easy 

to  decide  upon  a  scale  for  A,  B,  and  C.  This  sort  of  inconsistency  is 

common  in  human  judgments,  especially  when  complicated  issues  are 
involved. 

Thomas  Saaty  of  the  University  of  Pennsylvania  has  developed  a 
matrix  eigenvector  procedure  for  constructing  scales  of  merit  based  on 
inconsistent  pairwise  comparisons.  The  method  has  been  applied  in  a 
wide  variety  of  planning  and  decision  problems. 

This  report  presents  an  alternative  approach  that  is  preferable 
to  the  eigenvector  procedure  in  several  important  respects.  The 
proposed  procedure  is  derived  within  a  statistical  framework  and  is 

compared  with  the  eigenvector  method  on  the  basis  of  theoretical  and 

empirical  considerations. 

The  remainder  of  this  introduction  discusses  the  motivation  for 
dealing  formally  and  quantitatively  with  subjective  information  and 
provides  a  brief  review  of  some  of  the  relevant  literature.  Section 
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II  provides  a  short,  nonrigorous  discussion  of  the  eigenvector  method 
and  the  proposed  method  for  utilizing  subjective  judgments  in  quanti¬ 
tative  analysis.  An  example,  illustrating  the  use  of  the  two  methods 
as  well  as  similarities  and  differences  in  their  results,  is  intro¬ 
duced  in  this  section  and  examined  throughout  the  report. 

In  Section  III  we  give  rigorous  definitions  and  develop  a  frame¬ 
work  for  treating  the  estimation  problem  with  classical  statistical 
techniques.  Section  IV  provides  a  mathematical  treatment  of  the 
eigenvector  method.  Section  V  deals  with  the  application  of  subjec¬ 
tive  judgment  methods  to  the  study  of  hierarchical  structures.  The 
example  introduced  in  Section  II  is  considered  in  further  detail 
there. 

Section  VI  introduces  the  geometric  mean  vector  and  gives 
theoretical  justification  for  its  use  as  an  estimator  of  subjective 
scales.  In  Section  VII  we  define  a  statistical  measure  of  consistency 
for  subjective  judgment  matrices.  Section  VIII  presents  results 
of  a  Monte  Carlo  study  comparing  the  two  methods.  Section  IX  con¬ 
siders  in  greater  detail  the  example  introduced  in  Section  II  and 
expanded  on  in  Section  V. 

QUANTITATIVE  ANALYSIS  OF  SUBJECTIVE  DATA 

The  quantification  of  subjective  data  is  essential  for  dealing 
with  a  wide  class  of  problems  whose  solution  by  other  methods  would  be 
extremely  difficult  or  impossible.  Such  problems  are  often  amorphous 
and  vaguely  stated.  They  involve  large,  multifaceted  issues  of  impor¬ 
tance  to  decisionmakers  and  interest  groups  with  diverse  backgrounds 
and  biases.  Their  outcomes  may  determine  the  allocation  of  large 
sums  of  public  money  and  impinge  critically  on  the  public  interest. 
Moreover,  some  facets  of  the  problems  may  lack  any  well-defined, 
scalar-valued  measures  of  merit.  Even  if  there  are  appropriate  mea¬ 
sures,  the  collection  of  relevant  objective  data  might  be  prohibitively 
expensive  or  impossible. 

Such  problems  frequently  arise  in  the  assessment  of  future  needs 
for  large  organizations.  As  an  example,  consider  the  problem  of  long- 
range  planning  in  the  U.S.  Air  Force.  This  problem  involves  a  great 


many  interrelated  issues:  the  effects  of  political  and  economic 
factors  on  national  security,  the  importance  of  various  geographic 
regions  to  U.S.  interests,  the  threat  posed  by  conflicts  of  different 
types  in  different  regions,  the  current  strength  of  forces  to  deal 
with  such  conflicts,  and  so  on.  Although  it  might  be  possible  to 
define  objective  yardsticks  to  deal  with  some  of  Lhese  issues,  it 
certainly  is  not  possible  for  all  of  them.  For  some  issues,  sub¬ 
jective  judgments  of  relative  importance  or  value  are  the  only 
measures  available. 

In  some  problems  the  best  information  available  is  subjective,  so 
why  is  quantitative  analysis  desirable  at  all?  Why  not  just  ask  the 
experts  to  make  plans  and  decisions  based  on  an  informal,  intuitive 
analysis?  In  fact,  problems  that  are  not  amenable  to  hard  analysis 
are  frequently  resolved  through  intuitive  analysis  by  experts  and 
decisionmakers.  However,  there  are  several  good  reasons  for  using  a 
formal,  quantitative  approach  in  these  problems. 

A  formal  analytic  framework  gives  structure  and  definition  to 
an  amorphous  mass  of  data.  It  allows  the  decisionmaker  to  consider 
relevant  information  systematically  and  to  examine  options  and  con¬ 
sequences  one  at  a  time.  In  such  a  framework,  the  analyst  can  break 
an  unmanageable  problem  into  manageable  parts  and  then  synthesize 
information  about  the  parts  in  a  rational  fashion. 

A  formal  analytic  framework  also  permits  sensitivity  analysis  on 
alternative  judgments.  When  a  problem  is  considered  within  a  formal 
framework,  tradeoffs  among  alternative  judgments  can  be  spelled  out 
explicitly,  and  the  effects  of  variations  in  subjective  judgments  on 
outcomes  can  be  studied.  Sensitivity  analysis  may  even  provide  a 
basis  for  resolving  different  points  of  view. 

Perhaps  the  greatest  advantage  of  a  formal  analysis  in  government 
and  military  applications  is  that  it  is  repeatable.  Formal  analysis 
provides  the  audit  trail  that  is  so  important  in  matters  involving 
extensive  allocation  of  public  resources  and  impinging  on  the  public 
interest . 

Research  literature  on  the  use  of  subjective  information 
emphasizes  three  major  issues:  how  to  elicit  meaningful  subjective 
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judgments  from  individuals  or  groups,  how  to  synthesize  subjective  and 
objective  data  obtained  from  various  facets  of  a  large  problem,  and 
how  to  construct  measurement  scales  based  on  subjective  information. 
Following  is  a  review  of  some  of  the  literature  related  to  each  of 
these  issues. 

ELICITING  SUBJECTIVE  JUDGMENTS 

Methods  for  eliciting  subjective  judgments  have  received 
considerable  attention  in  operations  research  and  forecasting 
literature.  Two  such  methods  are  war  gaming  and  scenario  writing, 
both  of  which  are  used  extensively  in  military  planning  to  provide 
insights  into  possible  future  environments  and  needs. 

Much  of  the  literature  on  eliciting  judgments  deals  with  the 
problem  of  acquiring  a  collective  expert  opinion  free  from  the  usual 
negative  effects  of  group  pressure.  An  important  method  in  this 
category  is  the  Delphi  technique,  a  controlled  feedback  procedure 
originally  developed  by  researchers  at  The  Rand  Corporation  [1].  In 
Delphi,  a  researcher  interrogates  a  group  of  experts  individually 
concerning  their  opinions  on  possible  future  events.  The  researcher 
assembles  means  and  quartiles  for  quantitative  data  thus  obtained  and 
presents  them  individually  to  group  members  along  with  arguments  and 
comments  made  by  individuals.  Group  members  can  then  revise  their 
judgments.  The  procedure  is  repeated  until  the  range  of  judgments 
narrows.  The  controlled  feedback  mechanism  in  Delphi  makes  it  pos¬ 
sible  for  a  group  of  experts  to  avoid  the  usual  social  pressures  of 
open  discussion.  The  method  has  been  used  in  many  military  and 
industrial  applications  (see,  e.g.,  (2j,  (3]). 

The  Delphi  technique  has  given  rise  to  a  number  of  modifications. 
The  Probe  method  designed  by  researchers  at  TRW  for  forecasting  tech¬ 
nological  events  combines  Delphi  with  a  timing  chart  structure  so 
that  events  can  be  considered  in  sequence  [4].  The  method  of  qual¬ 
itative  controlled  feedback  proposed  by  Press  [5]  is  similar  to 
Delphi  in  that  it  uses  a  controlled  feedback  loop  to  aid  groups  in 
arriving  at  judgments,  but  it  differs  in  that  at  each  iteration,  mem¬ 
bers  are  supplied  only  arguments  and  comments  from  the  group,  with  no 
information  about  the  quantitative  distribution  of  group  answers. 


SYNTHESIZING  DATA  IN  LARGE  PROBLEMS 


Economists  and  statisticians  have  proposed  a  variety  of  methods 
for  breaking  large  problems  into  smaller  pieces  and  quantitatively 
synthesizing  subjective  and  objective  data  from  the  pieces.  One  of 
the  most  popular  of  these  methods  is  multi-attribute  utility  theory, 
which  provides  a  framework  for  selecting  an  optimal  decision  from 
among  multiple  alternatives  when  some  effects  of  the  decision  can  be 
measured  only  subjectively.  The  expected  value  of  each  alternative  is 
determined  as  a  function  of  the  decisionmaker's  preferences  for  the 
possible  consequences  and  the  probabilities  that  the  alternative  will 
lead  to  those  consequences.  The  probabilities  are  generally  deter¬ 
mined  from  subjective  judgments.  Some  of  the  decisionmaker's  pref¬ 
erences  are  determined  on  the  basis  of  subjective  indexes  such  as 
aesthetic  appeal,  and  others  are  determined  on  the  basis  of  objective 
measures  such  as  cost.  The  alternative  with  maximum  expected  Value  is 
chosen  as  the  optimum  decision. 

The  mathematical  foundation  for  mult i -attribute  utility  theory 
was  laid  by  von  Neumann  and  Morgenstern  [6].  Application  of  the 
theory  to  business  problems  was  pioneered  by  Raiffa  and  extended  by 
Keeney  and  others.  The  theory  has  been  applied  to  many  problems  in 
industrial,  government,  and  military  settings  (see  (7),  [8],  [9], 

[  1 0 1 ,  [11]).  The  1976  book  by  Keeney  and  Raiffa  gives  an  excellent 
treatment  of  the  subject  [12]. 

A  similar  method  was  applied  to  military  problems  in  a  1958 
Master's  thesis  by  Wells,  who  gives  a  detailed  framework  for  assessing 
the  relative  desirability  of  existing  or  proposed  weapon  systems. 
System  desirability  is  determined  as  a  function  of  feasibility,  cost, 
and  an  attribute  Wells  calls  "military  worth."  Wherever  possible, 
objective  measures  are  used  to  evaluate  these  three  factors,  and 
expert  judgments  are  used  where  there  are  no  objective  measures.  In 
particular,  military  worth  is  an  aggregate  property  evaluated  by  ana¬ 
lyzing  a  complex  hierarchy  and  subjective  scales  for  a  number  of  vari¬ 
ables.  The  Honeywell  Corporation  used  Wells’s  method  in  a  military 
planning  model  called  PATTERN  [13].  A  detailed  description  of  the 
method  can  be  found  in  Ref.  [14]. 
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Saaty  proposes  that  complex  decision  problems  be  viewed  in 
terms  of  hierarchies  of  objects  or  properties.  At  each  level  of  a 
hierarchy,  Saaty  uses  subjective  judgments  to  estimate  a  merit  scale 
of  the  objects.  Scales  from  all  the  levels  are  combined  mathemati¬ 
cally  to  provide  quantitative  information  about  the  whole  problem. 

The  results  in  this  report  are  applied  within  the  framework  of  hier¬ 
archical  analysis  proposed  by  Saaty.  Hierarchies  are  discussed  in 
Section  V. 

CONSTRUCTING  MEASUREMENT  SCALES 

Many  methods  have  been  developed  for  constructing  scales  of 
measurement  based  on  subjective  data.  Several  books  and  hundreds  of 
articles  have  been  written  about  these  methods.  A  classic  reference 
for  early  contributions,  especially  for  work  on  psychophysical  scales, 
is  Torgerson's  1958  book  [15]. 

Churchman  and  Ackoff  did  pioneering  work  in  the  area  of  esti¬ 
mating  scales  of  values  for  decision  problems  in  1954.  They  used  a 
criterion  of  additive  order  consistency  to  estimate  scales  from  suc¬ 
cessive  subjective  judgments.  Their  1954  paper  described  a  number 
of  applications  to  industrial  problems  [16].  Wells  and  others  later 
applied  the  Churchman  and  Ackoff  method  in  military  decision  problems 
[14]. 

Much  of  the  literature  on  subjective  scales  concerns  the  esti¬ 
mation  of  scales  from  pairwise  comparison  data.  A  good  deal  of  sta¬ 
tistical  work  in  this  area  goes  under  the  name  "paired  comparisons." 

In  the  simplest  paired  comparison  experiment,  each  of  several  judges 
examines  a  number  of  objects  two  at  a  time  and  states  which  of  the 
two  objects  is  preferred.  No  indication  of  strength  of  preference  is 
given.  Data  from  these  paired  comparisons  are  then  used  in  a  statis¬ 
tical  model  to  estimate  a  scale  of  preference  for  the  objects.  Such 
an  experiment  might  be  used  by  marketing  researchers  to  determine  the 
relative  taste  appeals  of  several  new  food  items. 

A  good  reference  for  the  statistical  theory  of  paired  comparisons 
is  David's  1963  book  [17].  A  bibliography  of  recent  articles  on  the 
subject  was  compiled  in  1976  by  Davidson  and  Farquhar  [18]. 
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Saaty  has  proposed  another  method  for  estimating  subjective 
scales  using  pairwise  comparisons  in  which  a  single  judge  makes 
pairwise  comparisons  of  a  number  of  objects.  For  each  pair,  the  judge 
states  not  only  which  object  is  preferred,  but  to  what  degree  that 
object  is  preferred  over  the  other.  A  preference  scale  is  determined 
for  the  objects  based  on  an  eigenvector  analysis  of  the  matrix  of 
pairwise  comparisons. 

Saaty  has  published  a  number  of  articles  (Refs.  [19]  to  [30]) 
describing  the  eigenvector  procedure  for  estimating  subjective  scales 
and  illustrating  the  usefulness  of  this  procedure  in  analyzing  complex 
hierarchical  structures.  He  has  applied  the  procedure  in  a  broad 
range  of  problems  in  the  social  sciences  ([19],  [21],  [28]).  The 
procedure  has  also  gained  acceptance  in  military  applications  and  is 
currently  being  used  as  a  tool  in  Air  Force  long  range  planning. 
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II.  PAIRWISE  COMPARISONS,  THE  JUDGMENT  MATRIX,  AND 
THE  ESTIMATION  PROBLEM 


Consider  the  problem  of  purchasing  a  new  car.*  Suppose  that  a 
preliminary  investigation  yields  five  specific  makes  that  seem  appro¬ 
priate.  The  price  of  each  make  is  known,  and  although  some  other 
measures  of  merit  may  have  been  quantified  (principally  performance 
measures),  the  important  subjective  question  of  how  much  each  car 
satisfies  the  overall  needs  is  difficult  to  quantify. 

We  will  attempt  to  assign  to  each  make  of  automobile  an  estimate 
of  utility  in  such  a  way  that  if  u^  is  the  utility  of  the  ith  make, 
then  uj/uj  is  3  measure  of  the  preference  of  the  ith  make  to  the  jth 
make.  The  vector  (Uj ,  u,,...,^)  will  be  called  a  ratio  scale. 

Some  aspects  of  the  usefulness  of  such  a  ratio  scale  are  imme¬ 
diately  apparent.  We  could,  in  this  example,  choose  between  the  cars 
on  the  basis  of  utility  per  dollar  of  initial  cost,  or  with  more  fore¬ 
sight,  on  the  basis  of  utility  per  dollar  of  expected  life  cycle  cost. 

To  estimate  the  vector  of  utilities  (u^,  Saaty  has 

suggested  the  following  procedure  ([19]  to  [30],  esp.  [28]):  We 

construct  a  matrix  composed  of  our  subjective  estimates  of  the  ratios 

of  the  utilities  of  all  possible  pairwise  combinations,  so  that  the 

elements  a.  .  of  the  matrix  A  are  our  estimates  of  u./u..  Thus  we 
x.J  i  J 

know  that  the  diagonal  elements  are  given  by  a .  .  =1,  i=l,...,5. 

X  T  * 

Additionally,  the  lower  off-diagonal  elements  are  determined  by 
the  upper  off-diagonal  elements:  a^  ^  =  1/a^  ^ . 

Saaty  [28]  proves  that  in  this  case  the  matrix  A  has  a  maximal 
eigenvalue  and  a  corresponding  eigenvector  (the  dominant  eigenvector) 
all  of  whose  components  are  positive.  Saaty  proposes,  primarily  with 


This  example,  due  to  Capt.  Jordan  Kreindler  and  Major  Michael 
Parmentier  of  the  U.S.  Air  Force  Directorate  for  Programs  Evaluation, 
Systems  Analysis  Division,  is  treated  in  further  detail  in  Sections  V 
and  IX. 


empirical  justification,  that  this  dominant  eigenvector  be  used  as  an 
estimate  of  the  ratio  scale. 


Suppose  that  when  we  form  our  estimates,  the  relative  utility  of 
Make  1  to  Make  2  is  2,  of  Make  1  to  Make  3  is  1/9,  of  Make  1  to  Make  4 
is  1/6  and  of  Make  1  to  Make  5  is  1/7.  Then  the  first  row  of  our 
judgment  matrix  has  the  form 


1,  2,  1/9,  1/6,  1/7  . 


Continuing,  suppose  that  we  have  filled  in  the  upper  off-diagonal  of 
our  judgment  matrix: 


1  2  1/9  1/6  1/7 

1  1/9  1/6  1/7 

1  6  4 

1  1/7 

1 


Then,  in  view  of  rec 

iprocal  i 

symmetry 

we  have 

1 

2 

1/9 

1/6 

1/7 

1/2 

1 

1/9 

1/6 

1/7 

A  =  9 

9 

1 

6 

4 

6 

6 

1/6 

1 

1/7 

7 

7 

1/4 

7 

1 

Continuing  with  this  example  we  compute  the  dominant  eigenvector 
w  of  this  matrix  and  get: 


.0378 
.0294 
w  =  .5239 

.1131 
.2958 


Thus,  in  this  case,  our  estimate  of  the  utility  of  the  first  make 
is  .0378  and  of  the  third  make  is  .5239.  For  a  detailed  treatment  of 
this  procedure  see  Refs.  [19]  and  [28]. 
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This  example  is  discussed  in  more  detail  in  Sections  V  and  IX, 
where  it  is  expanded  to  illustrate  the  value  of  ratio  scales  in 
analyzing  hierarchical  structures.  In  application  to  hierarchies  it 
is  assumed  that  the  objects  at  each  level  of  the  hierarchy  depend  on 
the  objects  of  the  next  lower  level  in  some  way.  The  procedure 
enables  the  user  to  estimate  the  influence  each  object  in  a  level  has 
on  all  the  objects  or  goals  in  superior  levels. 

For  problems  where  the  eigenvector  procedure  is  useful  there  is 
another  estimation  procedure  that  is  preferable  in  several  respects. 

Where  Saaty  would  estimate  the  utility  of  the  ith  object  with 
the  ith  component  of  the  dominant  eigenvector,  we  give  arguments 
that  a  better  estimator  is  given  by  the  vector  v  =  v.,v„,...,v  , 
where 


v. 

1 


n 

=  n 

j=i 


l/n 


i,J 


is  the  geometric  mean  of  the  elements  in  the  ith  row  of  A.  In  the 
example  above  this  yields  the  following  estimates: 


Object 

Eigenvector 

Geometric 
Mean  Vector 

1 

.0378 

.0409 

2 

.0294 

.0310 

3 

.5239 

.5307 

4 

.1131 

.1132 

5 

.2958 

.2842 

Compared  with  the  dominant  eigenvector,  the  geometric  mean  vector 

* 

1.  Is  statistically  better; 

2.  Is  easier  and  faster  to  calculate  on  a  hand  calculator  or 
computer; 
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3.  Gives  rise  to  a  measure  of  consistency  that  is  more  meaning¬ 
ful  than  the  eigenvector-eigenvalue  measure; 

4.  Gives  rise  to  a  measure  of  consistency  that  has  known  statis¬ 
tical  properties,  allowing  tests  of  hypotheses,  confidence 
interval  estimation,  etc.; 

5.  Gives  rise  to  estimates  of  utility  with  known  statistical 
properties,  allowing  tests  of  hypotheses,  confidence  interval 
estimation,  etc.; 

6.  Is  supported  by  statistical  literature  describing  methods  of 
handling  a  wealth  of  variations  of  the  problem; 

7.  Is  rooted  in  a  mathematical  approach  to  estimation  that  pro¬ 
vides  an  intuitive  understanding  of  the  problem  and  a  means 
for  assessing  suitability  of  the  method. 
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III.  CONSISTENT  MATRICES  AND  RATIO  SCALES 


Consider  a  set  of  n  activities  or  objects  Ej,  ,  which 

contribute  to  some  objective.  Suppose  the  activities  can  be  ranked 

on  a  ratio  scale  (u, ,  u_,...,u  ),  u.  >  0,  so  that  u./u.  measures  the 

degree  to  which  E^  is  more  important  than  E^  relative  to  the  objective. 

In  particular,  u^/u^  >  I  if  E.  is  more  important  than  E ^ .  Let  A  = 

[a.  .]  be  the  n  x  n  matrix  of  pairwise  comparisons  of  E. ,  E,,...,E 
l ,  j  l  z  n 

given  by 


ai,j  =  ui/uj  ’  i,j  =  1*2>--*>n 


(3.1) 


Then  A  has  the  property  that 


*j,i  =  1/ai,j’  i,j  =  * 


(3.2) 


and  in  particular 


i.i 


-  1,  i  —  1,2,. ..,n. 


A  square  matrix  A  with  positive  entries  satisfying  (3.2)  will  be  called 
a  judgment  matrix. 

It  follows  immediately  from  (3.1)  that 


i .  .a .  ,  =  a .  . 

i.J  J,k  i.k 


(3.3) 


A  matrix  with  positive  entries  satisfying  (3.3)  is  said  to  be  consistent. 
It  is  easy  to  see  that  every  consistent  matrix  is  a  judgment  matrix. 

Let  A  be  an  arbitrary  consistent  matrix.  Because 


a.  .  =  a.  ,/a,  , 

J.k  l.k  1 ,  J 


for  any  j , k. 
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every  element  of  A  can  be  determined  from  the  first  row  of  A.  It 
follows  that  A  is  a  matrix  of  rank  one  with  exactly  one  nonzero 
eigenvalue.  Moreover,  it  follows  from  (3.3)  that 

A2  =  nA  . 


Thus  any  column  of  A  is  an  eigenvector  of  A,  and  the  single  nonzero 
eigenvalue  of  A  is  n. 

Let  w  =  Wj,  w2>..-,wn  be  any  eigenvector  corresponding  to 
the  eigenvalue  n.  For  any  k,  the  kth  column  of  A  is  an  eigenvector 
corresponding  to  the  same  eigenvalue;  therefore  for  each  i  and  j, 


w .  = 

l 


ca .  , 
i,k 


w . 
J 


ca 


for  some  c  ^  0,  and  therefore 


w. 

i 


Thus,  w  is  a  ratio  scale  for  A.  In  fact,  it  is  clear  that  there  are 
infinitely  many  such  scales,  each  one  corresponding  to  a  different 
scalar  multiple  of  the  kth  column  of  A. 

The  normalized  eigenvector  with  components 


w . 
i 


i=l 


is  the  particular  scale  that  Saaty  [23]  associates  with  the  consistent 
matrix  A. 


A  ratio  scale  corresponding  to  a  consistent  matrix  A  can  be 
derived  in  several  ways.  Any  column  of  A  is  such  a  scale.  The  vector 
of  reciprocals  of  elements  of  an  arbitrary  row  of  A  is  also  a  ratio 
scale  for  A.  It  is  easy  to  see  that  the  vector  r  of  row  sums  defined 
by 


h 


n 

1 


r .  =  x.  a . 

1  jnl  l.J 


and  the  geometric  mean  vector  v  defined  by 

1/n 


n 


v.  =  II  a. 

1  j-1  1,J 


also  provide  ratio  scales  for  A.  When  these  scales  are  normalized, 
they  are  equal  to  the  normalized  eigenvector  scale  for  a  consistent 
matrix . 
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IV.  SAATY' S  NORMALIZED  EIGENVECTOR  SCALE 


Consider  again  the  activities  E, ,  E„,...,E  that  contribute  to 
°  1  2  ’  n 

some  objective.  Suppose  a  judge  makes  pairwise  comparisons  on  some 
scale  of  the  relative  importance  of  each  pair  of  activities  with 


respect  to  the  underlying  objective.  If  a^  represents  the  relative 

importance  of  E.  over  E.,  so  that  a.  .  >  1  if  and  only  if  E.  is  more 
^  J  ^  >  J  i 

important  than  E  ^  ,  it  is  then  natural  to  insist  that  the  judge  make 

comparisons  in  such  a  way  that 


a .  .  =  1/a .  .  for  each  i ,  i  . 

i,J  J,i 


In  other  words,  such  a  pairwise  comparison  matrix  is  a  judgment  matrix. 
The  ideal  pairwise  comparison  matrix  would  also  be  consistent.  For 
example,  if  E ^  is  twice  as  important  as  E^  and  E^  is  three  times  as 
important  as  E^,  one  would  expect  E^  to  be  six  times  as  important  as 
E^.  However,  human  judgment  is  often  inconsistent,  and  it  is  not 
likely  that  a  judge  making  pairwise  comparisons  will  construct  a 
consistent  matrix  except  in  cases  where  the  dimension  is  small.  A 
simple  example  in  which  pairwise  comparisons  do  not  result  in  a  con¬ 
sistent  matrix  is  that  of  a  tournament:  X  may  win  against  Y  and  Y 
against  Z,  but  X  may  lose  to  Z. 

The  problem  we  consider  is  this:  Given  an  inconsistent  judgment 
matrix  A,  how  can  we  construct  a  ratio  scale  that  in  some  sense  best 


reflects  the  information  in  the  matrix?  Saaty  proposes  that  the 
appropriate  scale  is  the  normalized  eigenvector  corresponding  to  the 
maximal  eigenvalue  of  A. 

Saaty  [28]  argues  as  follows.  If  the  judgment  matrix  A  is  con¬ 
sistent,  then  the  normalized  eigenvector  corresponding  to  the  single 
nonzero  eigenvalue  n  does  give  the  underlying  ratio  scale:  A  theorem 
of  Frobenius  for  matrices  with  positive  entries  [31]  guarantees  that 
any  judgment  matrix  has  a  positive  eigenvalue  L  that  exceeds  all  the 
other  eigenvalues  in  absolute  value.  This  maximal  eigenvalue  has  an 
associated  eigenvector  that  is  positive  in  all  its  components. 


Now  an  inconsistent  judgment  matrix  can  be  viewed  as  having  been 
derived  from  a  consistent  one  by  perturbation  of  some  or  all  of  the 
matrix  components.  Because  the  eigenvalues  and  eigenvectors  of  a 
matrix  depend  continuously  on  its  components,  small  perturbations  in 
the  components  will  result  in  small  changes  in  the  eigenvalues  and 
eigenvectors.  Thus  when  the  perturbations  of  the  components  are 
small,  the  maximal  eigenvalue  is  close  to  n,  and  the  corresponding 
normalized  eigenvector  is  close  to  the  normalized  eigenvector  of  the 
unperturbed  consistent  matrix.  Therefore,  Saaty  selects  the  suitably 
normalized  eigenvector  associated  with  the  maximal  eigenvalue  as  the 
ratio  scale  corresponding  to  the  judgment  matrix. 

Saaty  also  proposes  an  index  of  consistency  for  judgment  matrices. 
He  shows  that  an  n  x  n  judgment  matrix  whose  only  nonzero  eigenvalue 
is  n  must  be  consistent,  and  that  the  maximal  eigenvalue  L  for  an 
inconsistent  judgment  matrix  is  strictly  greater  than  n.  Therefore 
he  uses  the  normalized  difference 


as  the  index  of  consistency  of  an  n  x  n  judgment  matrix  with  maximal 
eigenvalue  L.  Notice  that  the  index  is  zero  for  consistent  matrices 
and  positive  for  inconsistent  ones.  Saaty  also  shows  that  the  index 
increases  as  perturbations  of  the  components  away  from  consistency 
increase . 

Unfortunately,  the  question  of  how  small  the  perturbations  of 
matrix  components  must  be  to  give  rise  to  a  given  deviation  in  the 
maximal  eigenvalue  is  a  delicate  one.  Saaty  [28]  describes  an 
empirical  investigation  of  this  question  in  which  he  determines  the 
consistency  indices  corresponding  to  randomly  generated  judgment 
matrices  of  different  dimensions.  However,  because  the  eigenvector 
does  not  fit  into  any  standard  statistical  framework,  there  is  no 
readily  available  device  against  which  deviations  from  consistency 
can  be  measured. 
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Saaty  does  show  ([28,  p.  238])  that  the  consistency  index  p 
reflects  the  variance  in  judgmental  errors  for  an  inconsistent  matrix, 
in  the  following  sense.  Suppose  that  the  pairwise  comparisons  ^ 
in  the  judgment  matrix  A  actually  arise  from  perturbations  of  the 
ratios  of  components  of  some  underlying  scale  (u^,  u^,...^^),  i.e., 


u . 

i 

- e .  . 

u .  1,1 

J 


=  1  +  d.  . 

i.J 


Saaty  shows  that  for  small  d.  . ,  2p  is  an  estimate  of  the  variance  of 

i,J 

the  d.  ..  Starting  from  this  estimate,  Saaty  develops  a  test  of  the 
1 1 J 

hypothesis  of  consistency  for  a  judgment  matrix  ([28],  p.  238). 

The  choice  of  a  scale  to  be  used  in  filling  in  a  pairwise  compari 
son  matrix  is  somewhat  arbitrary.  Because  humans  find  it  difficult  to 
rank  more  than  about  seven  objects  at  a  time,  Saaty  recommends  a  sub¬ 
jective  pairwise  comparison  scale  consisting  of  the  integers  from  one 
to  nine  together  with  their  reciprocals.  In  this  scale,  a  value  of  1 
is  assigned  to  pairs  of  objects  that  are  equally  important.  The  inte¬ 
gers  3,  5,  7,  and  9  are  associated  with  descriptive  words  (9  means 
"absolute  importance,"  5  means  "essential  or  strong  importance"),  and 
the  integers  2,  4,  6,  and  8  are  used  for  intermediate  values.  Recipro 
cals  of  integers  are  used  so  that  the  matrix  of  pairwise  comparisons 
is  a  judgment  matrix--that  is,  is  reciprocal  symmetric. 
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V.  APPLICATIONS  OF  THE  RATIO  SCALE 


Saaty  presents  numerous  applications  requiring  the  estimation  of 
ratio  scales  from  pairwise  comparison  information  ((19]  to  [30]).  He 
cites  examples  in  economics,  political  science,  and  transportation 
planning,  as  well  as  in  personal  planning  areas  such  as  choosing  a 
school  or  a  vacation  spot. 

One  of  the  most  interesting  and  useful  applications  of  the  ratio 
scale  is  in  the  study  of  hierarchical  systems.  A  hierarchy  is  a  col¬ 
lection  of  objects  grouped  according  to  levels.  Objects  at  a  given 
level  of  the  hierarchy  depend  oil  objects  at  lower  levels.  The  objects 
at  one  level  may  be  ranked  on  a  ratio  scale  according  to  their  impor¬ 
tance  relative  to  a  given  object  at  the  next  higher  level.  Thus  one 
may  construct  a  system  of  ratio  scales,  one  scale  for  each  level  rela¬ 
tive  to  every  object  in  the  next  level  up. 

Once  such  a  system  of  ratio  scales  has  been  constructed,  it  can 
be  used  to  study  interactions  among  all  levels  of  the  hierarchy.  For 
example,  in  a  hierarchy  consisting  of  three  levels,  we  may  determine 
the  ranked  importance  of  objects  on  the  lowest  level  relative  to  each 
object  in  the  highest  level.  Suppose  for  simplicity  that  the  highest 
level  consists  of  a  single  object,  that  the  second  level  has  n  objects, 
and  the  third  has  m  objects.  Let  (Wj ,  w£ , • ■ • ,wn)  be  the  ratio  scale 
that  reflects  the  importance  of  objects  in  the  second  level  relative 
to  the  single  object  in  the  first  level.  Now  the  objects  in  the  third 
(lowest)  level  may  be  ranked  on  a  ratio  scale  relative  to  each  object 
of  the  second  level.  Let 


(ul,j-  U2,j-"-’Um,j)’  J  = 


be  the  ratio  scale  for  the  third  level  relative  to  the  jth  object  of 
the  second  level.  Then  according  to  Saaty  the  importance  of  objects 
in  the  lowest  level  relative  to  the  highest  level  may  be  measured  by 
the  vector  ( v  ^ ,  v,  ,...,v^),  where 
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n 

v.  =  y  u.  .  w . ,  i  =  1 ,2, . . . ,m  . 
i  A  i,J  J’  ’  ’ 

j=l 

In  matrix  notation,  if  w  =  (Wj ,w^ , . . . ,wn)  is  the  ratio  scale  for  the 
second  level  relative  to  the  single  object  in  the  first  level,  and  if 

ul,l  U1 ,2 

u2  ]  u2  2 

U  = 

u  ,  u  „ 

m,l  m ,  2 

is  the  matrix  whose  ^th  column  is  the  ratio  scale  for  the  objects  in 
the  third  level  relative  to  the  j^th  object  in  the  second  level,  then 

v  ~  Uw 

gives  a  scale  of  importance  of  objects  in  the  third  level  relative  to 
the  first. 

The  same  procedure  may  be  extended  to  hierarchies  with  more  than 
three  levels.  Thus  knowing  only  the  measures  of  importance  of  objects 
in  each  level  relative  to  individual  objects  in  the  adjacent  higher 
level,  we  may  deduce  their  ranked  importance  relative  to  objects  at 
all  higher  levels.  In  particular,  objects  at  the  lowest  level  can  be 
ranked  according  to  their  importance  relative  to  the  object  (or  objects) 
at  the  highest  level. 

As  an  example,  consider  the  problem  of  selecting  an  automobile. 
(This  example  is  treated  in  more  detail  in  Section  IX.)  The  problem 
can  be  viewed  in  terms  of  the  hierarchical  structure  shown  in  Fig.  1. 

The  highest  level  of  the  hierarchy  is  the  final  selection  of  the 
automobile.  On  the  second  level  are  attributes  of  the  automobi 1 es-- 
namely,  status,  cost,  economy,  and  size.  The  third  level  consists  of 
the  automobile  makes  to  be  considered.  The  automobiles  themselves  are 
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Fig.  1 — hierarchy  for  selecting  an  automobile 


ranked  according  to  each  of  the  attributes,  and  the  attributes  are 
ranked  according  to  their  importance  relative  to  the  overall  objective 
of  selecting  a  car. 

Table  1  gives  the  ratio  scales  determined  from  judgments  made  by 
one  prospective  buyer.  The  order  of  automobile  preference  for  this 
buyer  is  then  given  by  the  product  of  the  matrix  and  the  vector  in 
Table  1: 


0378 

.2703 

.3357 

.3143 

.0885 

.2847  H 

0294 

.4196 

.2929 

.4630 

.2425 

.3703  T 

5239 

.0554 

.1071 

.0596  . 

.2579  = 

.1119  M 

1131 

.  1571 

.  1500 

.  1288 

.4112 

.1397  D 

2958 

.0976 

.1143 

.0343 

.0934  C 
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Table  1 
RATIO  SCALES 


Ratio  Scale  of  Second  Level  Relative  to 
First  Level 


Status 

.0885 

Cost 

.2425 

Economy 

.2579 

Size 

.4112 

Ratio  Scales  of  Third  Level  Relative  to 
Attributes  at  the  Second  Level 


Status 

Cost 

Economy 

Size 

H 

.0378 

.2703 

.3357 

.3143 

T 

.0294 

.4196 

.2929 

.4630 

M 

.5239 

.0554 

.1071 

.0596 

D 

.1131 

.1571 

.1500 

.1288 

C 

.2958 

.0976 

.1143 

.0343 

The  final  ranking  reflects  the  buyer's  perception  of  the  relative 
status,  cost,  economy,  and  size  of  the  five  automobiles  considered 
as  well  as  his  judgment  of  the  relative  importance  of  these  four 
attributes  in  the  selection  of  an  automobile.  This  buyer's  first 
choice  should  be  T  and  his  last  choice  C. 


VI.  THE  GEOMETRIC  MEAN  SCALE 


It  is  clear  from  the  example  in  the  last  section  that  the  study 
of  interactions  among  various  levels  of  a  hierarchy  depends  heavily 
upon  our  assessment  of  the  ranked  importance  of  objects  at  each  level 
relative  to  objects  in  the  level  above.  The  basic  building  blocks  in 
a  hierarchical  study  are  the  ratio  scales  measuring  the  relative  impor¬ 
tance  of  objects  at  a  given  level.  Therefore  one  would  like  to  know 
that  the  estimates  of  the  ratio  scales  are  well-grounded  in  statistical 
theory,  that  they  work  well  empirically,  and  that  they  can  be  calcu¬ 
lated  quickly.  In  this  section,  we  propose  a  method  for  constructing 
a  ratio  scale  based  on  pairwise  comparisons  that  is  superior  to  the 
eigenvector  procedure  when  judged  according  to  each  of  these  three 
criteria. 

For  n  x  n  judgment  matrices  A  =  [a.  .]  and  C  =  [c.  .],  define 

1  >  J  t  >  J 

»(A,C)  =  /  £  j  (In  .t>J  -  In  c.  '  . 


It  is  not  difficult  to  verify  that  m  is  a  metric  on  the  space  of  n  x  n 
judgment  matrices.  Theorem  3  will  show  that  for  any  n  x  n  judgment 
matrix  A,  there  is  a  consistent  matrix  C  that  is  m-closest  to  A.  In 
fact,  such  a  consistent  matrix  is  given  by 


c 


it  j 


v./v. 
i  J 


» 


where 
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that  is,  vi  is  the  geometric  mean  of  the  elements  of  the  ith  row  of  A. 
We  will  use  the  vector  v,  suitably  normalized,  as  the  estimate  of  the 
ratio  scale  corresponding  to  A. 

The  following  two  invariance  properties  show  that  m  is  a  suitable 
choice  of  metric  for  the  space  of  judgment  matrices.  Their  proofs  are 
straightforward  and  will  not  be  provided  here. 

Theorem  1  (Invariance  under  Transpose),  (i)  Let  A  =  [a.  .]  and 

T  t  1 

C  =  |c.  .]  be  n  x  n  judgment  matrices.  Then  A  and  C  are  also  judg- 
1  f  J 

ment  matrices,  and 


m(AT,CT) 


m(A,C) 


(ii)  Let  A  =  [a.  . ]  be  an  n  x  n  judgment  matrix,  and  suppose  that 
1 1 J 

C  =  [c.  .]  is  the*  consistent  matrix  that  is  m-closest  to  A.  Then 

T  1*J  T 

C  is  the  consistent  matrix  that  is  m-closest  to  A  . 

Theorem  2  (Invariance  under  Change  of  Scale),  (i)  Let  A  = 

[a.  .]  and  C  =  [c.  . )  be  n  x  a  judgment  matrices  and  (w, ,  w„,...,w  ) 
1 » J  1 » J  i  z  n 

a  ratio  scale.  Define 


A'  =  la.  .w./w.J 
!»J  i  J 


C'  =  [c.  .w./w.]  . 

i.J  i  J 

Then  A',  C'  are  judgment  matrices,  and 
m(A\C')  =  m(A,C) 

(ii)  Let  A,  C,  A',  C' ,  be  as  in  (i),  and  suppose  that  C  is  the  m- 
closest  consistent  matrix  to  A.  Then  C'  is  the  m-closest  consistent 
matrix  to  A' . 

If 

The  existence  and  uniqueness  of  C  are  guaranteed  by  Theorem  3. 
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Recall  that  we  seek  a  procedure  for  associating  ratio  scales  to 
judgment  matrices  in  such  a  way  that  the  ratio  scales  capture  the  sub¬ 
jective  information  inherent  in  the  corresponding  matrices.  Let  A  be 

an  n  x  n  judgment  matrix.  Let  C  =  (c.  . ]  be  a  consistent  matrix  that 

i,J 

is  m-closest  to  A,  and  suppose  that  v  =  (vpv2> . . .  ,vn)  is  a  ratio  scale 
for  B;  i.e. , 


c .  . 
i.J 


=  v./v. 

i  j 


We  choose  (v^ . ,vn)  as  the  estimator  of  the  ratio  scale  corres¬ 
ponding  to  A. 

Under  this  association,  Theorem  1  guarantees  that  the  scale 

(1/v.,  l/v„ , . . . , 1/v  )is  the  estimator  of  the  scale  corresponding  to 
■p  i  4  n 

A  .  This  is  a  natural  requirement,  because  if  v./v.  estimates  a.  ., 

1  J  1  >  J 

then  U/v./l/v.)  should  estimate  a.  ..  Theorem  2  guarantees  that  our 
1  J  J  > 1 

choice  of  ratio  scale  is  invariant  under  a  scale  change  in  the  judg¬ 
ment  matrix.  The  eigenvector  scale  does  not  satisfy  either  of  these 
requirements . 

The  following  theorem  guarantees  that  the  geometric  mean  scale 

gives  the  m-closest  consistent  matrix  to  any  judgment  matrix. 

Theorem  3.  Let  A  =  [a.  . ]  be  an  n  x  n  judgment  matrix.  Let 

*  *  J 

C  =  (c^  ^  be  the  consistent  matrix  given  by 


where  v^  is  the  geometric  mean  of  the  elements  of  the  ith  row  of  A; 
i.e.  , 


v . 
i 


n 

=  II 

j=1 


i.J 


1/n 

J 


i 


1,2,. 


,n  . 
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Then  m(A,C)  is  the  minimal  m-distance  from  A  to  any  n  x  n  consistent 
matrix. 

Proof.  For  any  consistent  matrix  C  =  fc.  .1.  we  can  write 

-  i,JJ’ 


: .  .  =  w . /w .  , 

i,J  i  J 


where  w  =  (w^ ,w^ , . . . ,w^)  is  a  ratio  scale.  Thus  we  seek  a  scale  that 
minimizes  the  expression 


n 

[In  a.  .  -  (In  w.  -  In  w.)]^. 
i=l  j>i  ‘  1>J  1  J 

Because  the  estimating  scale  need  be  known  only  up  to  a  scale  factor, 
we  may  normalize  by  imposing  the  side  condition 

n 

n  w.  =  1  . 

i=l 

Let 

y.  .  =  In  a.  .,  i,j  =  1,2,. ..,n  , 

1  >  J  1  9  J 

bA  =  In  i  =  1,2, . . . ,n  . 

Then  the  problem  is  to  minimize 

£  E  i»f>j  -  (bt  -  b.)]2 

i=l  j>i 


under  the  side  condition 


Z  bi  = 


Since 


y,  ,  =  -y,  i,j  =  1,2,. ..,n 

J  ,  A  1  *  J 


y.  .  =  0  , 

yi,x 


this  is  equivalent  to  minimizing 


1  Z  Z  ir,, j  -  <\  -  yi 

i=l  j=l 


under  the  side  condition 


E  »i  =  0  ■ 

1=1 


Now  S  is  convex  in  the  differences  b.  -  b.  and  therefore  convex  in  the 

i  J 

vector  b,  so  its  minimum  occurs  at  the  point  where 


-  =  0  for  i  =  1,2,. ..,n 

3b. 

l 


Consequently  the  m-distance  from  A  to  C  is  minimized  by  the  vector  v 
given  by 


v 


k 


n 

j=i 


M 


l/n 


This  completes  the  proof  of  Theorem  3. 

Recall  from  Section  III  that  if  the  matrix  A  is  consistent,  then 
the  normalized  geometric  mean  scale  is  equal  to  the  normalized  eigen¬ 
vector  scale.  Interestingly,  the  two  scales  are  always  the  same  for 
judgment  matrices  of  dimension  less  than  or  equal  to  three.  To  see 
this  in  the  case  n  =  3,  let 

lab 
A  =  1/a  1  c 

1/b  1/c  1  . 

Then  the  geometric  mean  vector  for  A, 


(ab) 


1/3 


v  =  (c/a) 


1/3 


(l/bc)1/3 


is  an  eigenvector  for  A  corresponding  to  the  eigenvalue 
L  =  1  +  (ac/b)1/3  +  (b/ac)1/3. 


Since  L  is  of  the  form 


1  +  x  +  1/x  , 

its  value  is  no  less  than  3,  the  dimension  of  A  (with  equality  in  the 
consistent  case).  Therefore  w  is  an  eigenvector  corresponding  to  the 
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maximal  eigenvalue  for  A.  Hence  in  the  case  n  =  3,  the  normalized 
geometric  mean  vector  and  the  normalized  eigenvector  are  the  same. 

This  result  does  not  hold  in  general  for  inconsistent  matrices  with  n 
greater  than  3. 

We  have  shown  that  given  an  arbitrary  judgment  matrix  A,  the 
geometric  mean  vector  gives  rise  to  the  ra-closest  consistent  matrix  to 
A.  The  problem  of  representing  a  judgment  matrix  by  a  ratio  scale  can 
also  be  cast  in  the  framework  of  the  general  linear  model.  We  will 
show  that  under  suitable  assumptions  the  geometric  mean  vector  is  the 
maximum  likelihood  estimator  for  the  ratio  scale  corresponding  to  a 
judgment  matrix. 

Let  A  =  [ a ^  ^ ]  be  an  n  x  n  judgment  matrix.  We  assume  that  there 
is  an  underlying  scale  (w^,  whose  ratios  are  perturbed  to 

give  the  elements  of  A;  namely, 


w. 

l 

a .  .  =  e .  .  , 

i,J  Wj  i.J 


(6.1) 


and  thus 


In  a.  .  =  In  w .  -  In  w .  +  In  e .  .  , 
i.J  i  J  i,J 

i  =  1 ,2, . . . ,n;  j  >  i  . 


(6.2) 


Suppose  that  the  errors  e.  .  are  independent  and  lognormally  distrib- 

i , J  2 

uted  with  means  0  and  variances  (J  .  Making  the  substitutions 


Y 


In 

In 


In  Wj 


B  = 


E 


In 

In 


e 


e 


1,1 

1,2 


In 


a 


n-1  ,n 


In  w 


In  e 


n- 1  ,n 


i  ■  ■  ...  fr'il 
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we  have  the  general  linear  equation 
Y  =  XB  +  E 

where  the  matrix  X  has  components  -1,  0,  +1  determined  by  the 
equations  (6.2).  In  this  framework  it  is  well  known  [32]  that  the 
maximum  likelihood  estimate  for  B  =  [In  w.)  is  the  least-squares 
estimate  given  by 


-  £  In  a  , 

*  J*« 

and  that  the  estimate  has  all  of  the  usual  desirable  properties  of 
least-squares  estimates  under  the  general  linear  hypothesis.  Taking 
exponentials  we  obtain  the  maximum  likelihood  estimate  of  w^: 


b. 

l 


w.  =  exp(b) 


n 

n 

j=i 


ai,j 


1/n 


the  same  estimate  derived  above  from  the  metric  m  on  ttfe  space  of 
judgment  matrices. 

The  procedure  outlined  above  can  be  modified  to  solve  more 

general  estimation  problems.  For  example,  suppose  that  instead  of 

a  single  comparison  for  each  pair  of  objects  and  E^  we  have 

n.  .  comparisons,  where  n.  .  may  be  zero  (reflecting  missing  data) 

1  >  J  1  >  J 

or  greater  than  one  (reflecting  multiple  comparisons,  say  by  different 
judges).  The  problem  is  then  to  find  a  w  that  minimizes  the  sum  of 
squares: 


n  n.  . 

i.J 

s  =  23  ]C  aC  Un  ai,j,k  “  ^ln  wi  '  ln  wj^ 

i=l  j>i  k=l 


This  generalization  does  not  yield  a  simple  closed-form  solution  such 
as  the  geometric  mean  vector,  but  in  practice  S  can  be  minimized  using 
standard  least-squares  regression  packages. 
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VII.  THE  STATISTICAL  MEASURE  OF  CONSISTENCY 


The  geometric  mean  estimation  procedure  outlined  in  Section  VI 
leads  to  a  natural  measure  of  consistency  for  judgment  matrices  that 
is  well-grounded  in  statistical  theory  and  suited  for  use  in  statis¬ 
tical  hypothesis  testing. 

Let  A  be  an  n  x  n  judgment  matrix  whose  components  arise  from 
multiplicative  perturbations  of  ratios  of  an  underlying  scale  w-- 
that  is 


a 


i.j 


w . 
1 


where  In  e.  i  <  j  are  independent  random  variables  with  mean  zero 
1 » J  2 

and  variance  a  .  It  was  shown  in  Section  VI  that  the  maximum  likeli¬ 
hood  estimate  of  w^  is  the  geometric  mean  of  the  elements  of  the  ith 
row, 


/N. 

w. 

1 


a 

II 

j  =  l 


t 


Let  s  be  the  residual  mean  square 


2 

s 


d.f. 


where 
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n 

ss2  =  E  E  in  ai,j  ■ in  (wi/wj)2 . 
i=l  j>i 

d.f.  =  nC|li)  .  (n  .  d  =ln1llin12l 

2  2 

Then  s  is  an  unbiased  estimator  of  a  and  is  a  natural  measure  of 
consistency  of  A. 

2 

Recall  from  Section  VI  that  SS  can  be  viewed  as  the  squared 

2 

distance  from  A  to  the  m-closest  consistent  matrix.  Therefore  s 

is  zero  when  A  is  consistent,  is  close  to  zero  when  A  is  close  to 

consistent,  and  is  far  from  zero  when  A  is  far  from  consistent. 

2 

Moreover,  since  s  depends  entirely  on  ratios  of  elements  of  A,  it 
is  invariant  under  scale  changes. 


1 
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VIII.  EMPIRICAL  COMPARISON  OF  THE  TWO  METHODS:  A  MONTE  CARLO  STUDY 


The  geometric  mean  vector  gives  an  estimate  for  ratio  scales 
based  on  pairwise  comparison  data  that  is  easy  to  calculate,  satis¬ 
fies  the  theoretical  requirements  of  invariance  under  scale  change 
and  transpose,  and  is  well-grounded  in  statistical  theory.  It  follows 
from  Section  VI  that  under  certain  assumptions  on  the  distribution  of 
the  perturbations,  the  logarithms  of  the  geometric  mean  vector  compo¬ 
nents  are  the  minimum  variance  unbiased  estimators  of  the  logarithms 
of  the  underlying  ratio  scale  factors.  In  addition,  there  is  empirical 
evidence  that  the  geometric  mean  vector  is  a  better  estimator  of  ratio 
scales  than  the  eigenvector  when  perturbations  are  not  lognormally  dis¬ 
tributed,  particularly  in  cases  of  extreme  data  inconsistency.  This 
section  describes  results  of  a  Monte  Carlo  study  comparing  the  geometric 
mean  vector  and  the  eigenvector  as  estimators  of  ratio  scales. 

Suppose  we  start  with  an  n  x  n  consistent  matrix  A  with  elements 


Let  (e.  . ,  i  =  l,...,n,  j  )  i)  be  a  collection  of  positive  independent 
1  y  J 

random  variables  drawn  from  a  suitable  population,  and  construct  a 
perturbed  matrix  D  with  elements 


Then  D  is  a  judgment  matrix,  but 
not  necessarily  consistent.  Now 


for  i  =  1 ,2, . . .  ,n,  j  >  i 

for  i  =  1 ,2, .  .  .  ,n,  j  <  i 

for  i  =  1 ,2, . . . ,n,  j  =  i 

because  of  the  perturbations  it  is 
one  would  hope  that  any  estimate  of 


1 


J 
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an  underlying  ratio  scale  for  D  would  give  rise  to  a  consistent  matrix 
that  is  in  some  sense  close  to  A.  Two  measures  of  closeness  are  used 
in  the  Monte  Carlo  study  described  here:  The  sum  of  squares  of  errors 

n 

SSE  =  ^  “  ui^2 

i=l 

and  the  sum  of  squares  of  the  errors  of  logarithms. 

n 

SSL  =  ^2  [In  > 

i=l 

where 

(ui,  i  =  1 ,2, . . . ,n) 


is  the  actual  normalized  ratio  scale,  and 


(IT',  i  =  1 ,2, .  .  .  ,n) 


is  the  estimated  ratio  scale. 

The  choice  of  population  from  which  the  perturbing  random  vari¬ 
ables  e.  .  are  drawn  should  reflect  the  fact  that  in  a  judgment  matrix 
*  >  J 

constructed  by  an  actual  judge,  errors  are  likely  to  be  reciprocal 

symmetric.  That  is,  for  any  a,b  >  0,  the  random  variables  e.  .  should 

1 » J 

satisfy: 


P (a  <  e.  .  <  b) 
1  »  J 


=  P{a  < 


e .  . 

i  ,J 


<  b) 


Two  convenient  and  quite  dissimilar  distributions  satisfying  this 
property  are  a  lognormal  distribution  whose  underlying  normal  distri- 
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bution  has  mean  zero  and  a  distribution  obtained  from  the  ratio  of 

two  independent,  uniformly  distributed  random  variables.  We  consider 

perturbations  e.  .  drawn  from  populations  with  distributions  of  both 
1 » J 

of  these  types. 

Table  2  gives  results  of  the  Monte  Carlo  comparison  of  the  geo¬ 
metric  mean  vector  and  the  eigenvector  estimates  when  the  underlying 
ratio  scale  is  perturbed  by  lognormal  factors.  For  the  first  row, 

ratios  of  a  normalized  scale  consisting  of  five  elements  were  per- 

o 

turbed  by  e .  .  drawn  from  a  lognormal  population  with  o  =  .01.  Then 
i  >  J 

the  normalized  maximal  eigenvector  and  the  normalized  geometric  mean 
vector  for  the  resulting  judgment  matrix  were  computed.  This  proce¬ 
dure  was  repeated  in  1000  similar  trials,  each  time  with  the  same 
matrix  dimension  and  population  variance. 


Table  2 

MONTE  CARLO  COMPARISON  OF  GEOMETRIC  MEAN  VECTOR  AND  EIGENVECTOR 
FOR  RATIO  SCALES  PERTURBED  BY  LOGNORMAL  ERRORS 


;  r  !•>  1  .(i 

WIll>  ll  .<  t 


'.'•iint  -I 

1  rivr  .il  ! 


The  sums  of  squares  of  errors  and  sums  of  squares  of  errors  of 
logarithms  were  computed  for  individual  trials  and  then  totaled  over 
1000  trials  for  both  the  geometric  mean  vector  and  the  eigenvector 
estimates . 

Columns  3  and  4  in  the  table  give  the  total  sum  oi  squared  errors 
in  1000  trials.  Column  5  gives  the  percentage  of  trials  in  which  the 
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geometric  mean  vector  is  a  better  estimator  in  a  least  squares  sense 
then  the  eigenvector.  Columns  6,  7,  and  8  present  the  same  informa¬ 
tion  for  the  sum  of  squares  of  errors  of  logarithms. 

Comparisons  based  on  1000  trials  were  made  for  scales  consisting 
of  5,  7,  and  10  elements  with  perturbations  drawn  from  lognormal 
populations  with  underlying  normal  population  variances  of  .01,  .16, 
.64,  1.0,  and  2.0.  The  results  in  Table  2  indicate  that  the  eigen¬ 
vector  and  geometric  mean  vector  give  very  close  results  when  the  vari¬ 
ance  is  small--that  is,  when  perturbations  away  from  consistency  are 
minimal.  However,  for  larger  perturbations,  the  geometric  mean  vector 
deviates  less  from  its  underlying  ratio  scale. 

Table  3  gives  results  of  the  Monte  Carlo  comparison  of  the  two 
methods  when  the  underlying  ratio  scale  is  perturbed  by  random  vari¬ 
ables  drawn  from  a  population  of  ratios  of  uniform  random  variables. 

As  in  Table  2,  underlying  scales  with  5,  7,  and  10  elements  are  con¬ 
sidered  along  with  population  variances  for  In  e.  .  of  .01,  .16,  .64, 

*  *  J 

1.0,  and  2.0.  Again,  there  is  very  good  agreement  between  the  two 
estimators  when  variances  are  small.  But  for  larger  variances,  the 
geometric  mean  vector  is  closer  to  the  underlying  scale  in  both  the 
least  squares  and  the  least  squares  of  logarithms  sense. 


Table  3 

MONTF.  CARLO  COMPARISON  OF  GEOMETRIC  MEAN  VECTOR  AND  EIGENVECTOR 
FOR  RATIO  SCALES  PERTURBED  BY  RATIO  OF  UNIFORM  ERRORS 


t  T  l,  Mf.ll 


h.t 


We  conclude  that  in  the  presence  of  marked  inconsistency,  the 
geometric  mean  vector  provides  an  estimate  of  underlying  ratio  seal 
that  is  empirically  better  than  the  maximal  eigenvector. 
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IX.  EXAMPLE 


In  this  section  we  discuss  the  use  of  the  eigenvector  and  the 
geometric  mean  vector  in  a  specific  subjective  judgment  situation. 
Estimates  of  underlying  utility  vectors  and  consistency  values  derived 
from  the  two  methods  are  compared. 

Consider  the  automobile  selection  problem  introduced  in  Sections 
II  and  V.  The  hierarchical  structure  for  this  problem  is  given  in 
Fig.  1. 

Subjective  judgment  data  for  this  example  were  obtained  from  one 
prospective  buyer.  The  buyer  made  pairwise  comparisons  reflecting  his 
perceptions  of  the  relative  importance  of  the  attributes  of  status, 
cost,  economy,  and  size  in  selecting  an  automobile.  Judgments  were 
made  based  on  the  subjective  judgment  scale  developed  by  Saaty  (see 
p.  17).  The  buyer  made  comparisons  in  such  a  way  that  the  resulting 
pairwise  comparison  matrix  would  be  reciprocal  symmetric  (i.e.,  a 
judgment  matrix).  The  resulting  judgment  matrix  A  is: 


Status 

Cost 

Economy 

Size 

Status 

1 

1/5 

1/5 

1/2 

Cost 

5 

1 

1 

1/3 

Economy 

5 

l 

1 

1/2 

Size 

2 

3 

2 

1 

Next,  judgment  matrices  were  constructed  I rom  the  buyer's  pairwise 
comparisons  of  the  five  types  of  automobiles  relative  to  each  of  the 
two  attributes  of  status  and  sue: 
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Subjective  Comparison  Relative  to  Status 


H 

T 

N 

I) 

C 

H 

1 

2 

1/9 

1/6 

1/7 

T 

1/2 

1 

1/9 

1/6 

1/7 

M 

9 

9 

1 

6 

4 

=  11 

D 

6 

6 

1/6 

1 

1/7 

C 

7 

7 

1/4 

7 

1 

Subjective 

Compar i son 

Relative  to 

S  i  zv 

H 

T 

M 

D 

C 

H 

1 

1/2 

7 

3 

8 

T 

2 

1 

8 

4 

9 

M 

1/7 

1/8 

1 

1/3 

3 

=  C 

D 

1/3 

1/4 

1 

1 

4 

C 

1/8 

1/9 

1/3 

1/4 

1 

The  eigenvector 

and  the  geometric 

mean  vector 

were  calculated 

separately 

for  each 

of  the 

three  subjective  judgment  matrices 

above 

Resulting  scale  estimates 

for  the  two 

tochn i c|ues  are  given  in 

Table 

Not  1 ce 

that  the 

scales  determined 

by  the  two 

methods  arc- 

ve  ry 

close  in  value.  This  is  as  expected  from  theoretical  c ons i dera t i ons , 
because  the  two  methods  give  the  same  results  for  c ons i stent  matrices 
ami  should  agree'  closely  for  nearly  consistent  ones. 


42 


The  results  in  Table  4  indicate  that  the  prospective  buyer 
considers  autoaobile  sise  to  be  considerably  sore  important  than 
status,  cost,  or  econo ay.  Se  ranks  T  as  bavins  the  best  eize 
of  any  of  the  cars  under  consideration. 

Since  exact  values  were  available  for  cost  and  econony  of  the 
five  autonobile  types,  it  eras  not  necessary  to  conpute  scales  for  then 
fron  pairwise  conparisons.  The  vernalised  scale  values  for  the 
sutonohilos  relative  to  these  two  attributes  were  dete mined  to  be: 


Cost  Ratings 

Cceueuy  Ratings 

■ 

.2  >03 

.3337 

T 

.4190 

.2929 

a 

.0334 

.1071 

• 

.1371 

.1300 

c 

.0970 

.1143 

Pren  each  nethod,  we  now  have  scale  estinates  for  the  five  auto* 
nobile  types  relative  to  each  of  the  four  autonobile  attributes,  as 
well  as  a  scale  of  inportance  of  the  four  attributes.  Scale  estinates 
for  the  five  cars  relative  to  each  attribute  are  used  as  co lanes  of  a 
S  *  4  nstrix.  This  natrix  is  nultiplied  by  the  4*dinensiooal  vector 
of  inportance  of  attributes.  The  resulting  5*dionnsional  vector  re* 
fleets  the  prospective  buyer’s  ranking  of  the  five  autenobiles.  The 
calculations  for  both  nethods  are  carried  out  in  Table  5. 
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Tabic  S 

DETERMINATION  OF  RANKINGS  OP  FIVE  AUTOMOBILES 


Geoaetric  Mean  Vector  Method 


Make  Criterion  Final 

of  Car  Status  Cost  Econoay  Size  Scale  Scale 


3  . 


Eigenvector  Method 


Make 

of  Car 


Status 

Cost 

Econoay 

Size 

Criterion 


Final 

Scale 
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